DAOS-16661 common: Integrate PMDK logging system w/ VOS logging system. #14923

grom72 · 2024-08-13T20:28:40Z

pmemobj error and warning messages are written by default to stderr and syslog().
PMDK enables redirecting logging messages to user-specific functions.

All error and warning messages are reported using the DAOS logging function, to ensure a coherent events view for VOS and related pmemobj functions.

Requires: DAOS-16660 No error message on intentional pmemobj transaction abort pmdk-pkg#43 land to avoid error messages on intentionally canceled transaction.

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

github-actions · 2024-08-13T20:28:59Z

Ticket title is 'Integrate PMDK logging system w/ VOS logging system.'
Status is 'In Review'
https://daosio.atlassian.net/browse/DAOS-16661

daosbuild1 · 2024-08-14T08:35:14Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-14923/3/testReport/

To enable new message in log in NLT. Allow-unstable-test: true Priority: 2 Required-githooks: true Co-authored-by: Jan Michalski <[email protected]> Co-authored-by: Tomasz Gromadzki <[email protected]> Signed-off-by: Tomasz Gromadzki <[email protected]>

debian/changelog

To enable new message in log in NLT. Allow-unstable-test: true Priority: 2 Required-githooks: true Signed-off-by: Tomasz Gromadzki <[email protected]>

Priority: 2 Allow-unstable-test: true Skip-func-hw-test: false Required-githooks: true Signed-off-by: Tomasz Gromadzki <[email protected]>

Skip-nlt: true Priority: 2 Allow-unstable-test: true Skip-func-hw-test: false Required-githooks: true Signed-off-by: Tomasz Gromadzki <[email protected]>

brianjmurrell

@grom72 Packaging changes LGTM. Please re-request review from me once you have gotten your two +1s for the core code changes and I will +1 to remove my -1 review. I just don't want my +1 to be misunderstood to be for the core code changes also.

daosbuild1 · 2024-09-04T19:46:05Z

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14923/11/execution/node/1683/log

daosbuild1 · 2024-09-19T19:42:59Z

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14923/12/execution/node/635/log

Skip-nlt: true Priority: 2 Allow-unstable-test: true Required-githooks: true Signed-off-by: Tomasz Gromadzki <[email protected]>

daosbuild1 · 2025-01-16T18:09:07Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14923/36/execution/node/1349/log

daosbuild1 · 2025-01-16T18:10:28Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14923/36/execution/node/1396/log

daosbuild1 · 2025-01-17T16:22:36Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14923/37/display/redirect

daosbuild1 · 2025-01-17T22:20:43Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14923/38/execution/node/294/log

daosbuild1 · 2025-01-17T22:21:04Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14923/38/execution/node/297/log

daosbuild1 · 2025-01-17T22:37:20Z

Test stage Test RPMs on EL 8.6 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14923/38/display/redirect

Skip NLT tests for full HW validation Skip-NLT: true Skip-list: test_ior_intercept_libpil4dfs:DAOS-16260 test_daos_degraded_ec:DAOS-16503 test_dfuse_daos_build_wt_pil4dfs:DAOS-16732 Allow-unstable-test: true Priority: 2 Signed-off-by: Tomasz Gromadzki <[email protected]>

grom72 · 2025-01-21T07:12:02Z

https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14923/39/Fault_Injection/
The latest version of the PMDK warns when it is used in configuration without PMem.
It is expected behavior.
All other tests have passed:
https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14923/39/pipeline-graph/
https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14923/39/testReport/

jolivier23 · 2025-01-29T15:34:48Z

src/vos/pmdk_log.c

+pmdk_log_function(enum pmemobj_log_level level, const char *file_name, unsigned line_no,
+		  const char *function_name, const char *message)
+{
+	uint64_t dlog_prio = pmemobj_log_level_2_dlog_prio[level];


This is one example where pulling in PMDK into DAOS would be useful.

Anyway, this can be improved to be more efficient.

Add the following

#define _D_DEBUG_EXPANDED(func, flag, flag_var, ...)

Have _D_DEBUG call this with DD_FLAG(flag, D_LOGFAC) as the flag var.

In your case, you can have a map like the one you have above to each flag var

e.g. &DD_FLAG(DLOG_INFO, D_LOGFAC)

And instead of calling _D_DEBUG, call _D_DEBUG_EXPANDED with *flag_var you saved.

Additionally, one might consider passing these variables down to the lower level (e..g. register_debug_callback(callback, &DD_FLAG(...)). The check can just be if (*flag != 0) inside of pmdk and the upper level will take care of caching the value for the next call.

The _D_DEBUG_W_SAVED_MASK macro (instead of _D_DEBUG_EXPANDED) has been added and the pmdk_log_function() has been adjusted.

jolivier23

This can be improved

jolivier23 · 2025-01-29T16:24:54Z

#15810 for example

Co-authored-by: Jeff Olivier <[email protected]> Co-authored-by: Tomasz Gromadzki <[email protected]> Signed-off-by: Tomasz Gromadzki <[email protected]>

pmdk_log_function optimization based on _D_DEBUG_EXPANDED macro. Skip NLT tests for full HW validation Skip-NLT: true Skip-list: test_ior_intercept_libpil4dfs:DAOS-16260 Allow-unstable-test: true Priority: 2 Co-authored-by: Jeff Olivier <[email protected]> Co-authored-by: Tomasz Gromadzki <[email protected]> Signed-off-by: Tomasz Gromadzki <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Skip NLT tests for full HW validation Skip-NLT: true Skip-list: test_ior_intercept_libpil4dfs:DAOS-16260 Allow-unstable-test: true

jolivier23 · 2025-01-30T19:56:14Z

utils/build.config

@@ -28,5 +28,5 @@ ucx=https://github.com/openucx/ucx.git
 [patch_versions]
 spdk=https://github.com/spdk/spdk/commit/b0aba3fcd5aceceea530a702922153bc75664978.diff,https://github.com/spdk/spdk/commit/445a4c808badbad3942696ecf16fa60e8129a747.diff
 mercury=https://raw.githubusercontent.com/daos-stack/mercury/f3dc286fb40ec1a3a38a2e17c45497bc2aa6290d/na_ucx.patch
-pmdk=https://github.com/pmem/pmdk/commit/2abe15ac0b4eed894b6768cd82a3b0a7c4336284.diff
+pmdk=https://github.com/pmem/pmdk/commit/2abe15ac0b4eed894b6768cd82a3b0a7c4336284.patch,https://github.com/pmem/pmdk/commit/61e32285370e629e2b36bbb991b919e44f87d915.patch,https://github.com/pmem/pmdk/commit/518b7426a13b21f98b2d2c435fa645770899446a.patch


Is this same patch applied in the PMDK rpm builder?

@jolivier23, Patch https://github.com/pmem/pmdk/commit/2abe15ac0b4eed894b6768cd82a3b0a7c4336284.patch is not required for PMDK rpm builder as it is related to ARM architecture only.
The rest are the same.

grom72 · 2025-01-31T06:41:23Z

All tests passed except NLT and Fault injection due to new warnings reported by PMDK
https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14923/42/pipeline-console/
https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14923/42/pipeline-console/?start-byte=0&selected-node=1168#log-90
https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14923/41/VM_test/fileName.-1910336222/

grom72 force-pushed the grom72/pmdk-w-logging branch 3 times, most recently from 75504b7 to ad62bd1 Compare September 3, 2024 07:38

grom72 force-pushed the grom72/pmdk-w-logging branch from ad62bd1 to 149320c Compare September 3, 2024 10:59

grom72 marked this pull request as ready for review September 3, 2024 11:01

grom72 requested review from a team as code owners September 3, 2024 11:01

grom72 requested review from ashleypittman and jolivier23 September 3, 2024 11:15

brianjmurrell suggested changes Sep 3, 2024

View reviewed changes

debian/changelog Outdated Show resolved Hide resolved

HARK is reported as INFO

1011420

To enable new message in log in NLT. Allow-unstable-test: true Priority: 2 Required-githooks: true Signed-off-by: Tomasz Gromadzki <[email protected]>

grom72 requested a review from brianjmurrell September 3, 2024 20:56

Fix - force build

f9b926d

Priority: 2 Allow-unstable-test: true Skip-func-hw-test: false Required-githooks: true Signed-off-by: Tomasz Gromadzki <[email protected]>

grom72 added the release-2.6.1 Targeted for release 2.6.1 label Sep 4, 2024

Skip NLT tests as they blocks functional validation.

45e6548

Skip-nlt: true Priority: 2 Allow-unstable-test: true Skip-func-hw-test: false Required-githooks: true Signed-off-by: Tomasz Gromadzki <[email protected]>

brianjmurrell reviewed Sep 4, 2024

View reviewed changes

jolivier23 previously approved these changes Sep 5, 2024

View reviewed changes

grom72 added release-2.6.2 Targeted for release 2.6.2 and removed release-2.6.1 Targeted for release 2.6.1 labels Sep 6, 2024

grom72 requested a review from NiuYawei September 16, 2024 14:12

NiuYawei previously approved these changes Sep 18, 2024

View reviewed changes

Fix: apply git-format-clang chanhes

76fe5c6

Skip-nlt: true Priority: 2 Allow-unstable-test: true Required-githooks: true Signed-off-by: Tomasz Gromadzki <[email protected]>

grom72 requested review from jolivier23 and NiuYawei January 21, 2025 07:13

NiuYawei previously approved these changes Jan 22, 2025

View reviewed changes

grom72 requested review from ryon-jensen and brianjmurrell and removed request for brianjmurrell and ashleypittman January 23, 2025 21:18

jolivier23 reviewed Jan 29, 2025

View reviewed changes

jolivier23 requested changes Jan 29, 2025

View reviewed changes

jolivier23 added 2 commits January 30, 2025 09:29

Optimizing the use of the DD_FLAG() macro

1d3f89a

Co-authored-by: Jeff Olivier <[email protected]> Co-authored-by: Tomasz Gromadzki <[email protected]> Signed-off-by: Tomasz Gromadzki <[email protected]>

grom72 dismissed NiuYawei’s stale review via 13dd692 January 30, 2025 08:30

grom72 added 3 commits January 30, 2025 16:16

Unified parameters naming convention

2b2b30c

Signed-off-by: Tomasz Gromadzki <[email protected]>

pmemobj log to daos translation tab optimization

303a42d

Signed-off-by: Tomasz Gromadzki <[email protected]>

Merge remote-tracking branch 'origin/master' into grom72/pmdk-w-logging

ce9fa93

Cancel-prev-build: false Skip NLT tests for full HW validation Skip-NLT: true Skip-list: test_ior_intercept_libpil4dfs:DAOS-16260 Allow-unstable-test: true

grom72 requested a review from jolivier23 January 30, 2025 16:48

jolivier23 reviewed Jan 30, 2025

View reviewed changes

jolivier23 approved these changes Jan 30, 2025

View reviewed changes

grom72 requested review from NiuYawei and removed request for brianjmurrell January 30, 2025 20:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-16661 common: Integrate PMDK logging system w/ VOS logging system. #14923

DAOS-16661 common: Integrate PMDK logging system w/ VOS logging system. #14923

grom72 commented Aug 13, 2024 •

edited

Loading

github-actions bot commented Aug 13, 2024 •

edited

Loading

daosbuild1 commented Aug 14, 2024

brianjmurrell left a comment

daosbuild1 commented Sep 4, 2024

daosbuild1 commented Sep 19, 2024

daosbuild1 commented Jan 16, 2025

daosbuild1 commented Jan 16, 2025

daosbuild1 commented Jan 17, 2025

daosbuild1 commented Jan 17, 2025

daosbuild1 commented Jan 17, 2025

daosbuild1 commented Jan 17, 2025

grom72 commented Jan 21, 2025 •

edited

Loading

jolivier23 Jan 29, 2025

grom72 Jan 30, 2025

jolivier23 left a comment

jolivier23 commented Jan 29, 2025

jolivier23 Jan 30, 2025

grom72 Jan 31, 2025

grom72 commented Jan 31, 2025

DAOS-16661 common: Integrate PMDK logging system w/ VOS logging system. #14923

Are you sure you want to change the base?

DAOS-16661 common: Integrate PMDK logging system w/ VOS logging system. #14923

Conversation

grom72 commented Aug 13, 2024 • edited Loading

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented Aug 13, 2024 • edited Loading

daosbuild1 commented Aug 14, 2024

brianjmurrell left a comment

Choose a reason for hiding this comment

daosbuild1 commented Sep 4, 2024

daosbuild1 commented Sep 19, 2024

daosbuild1 commented Jan 16, 2025

daosbuild1 commented Jan 16, 2025

daosbuild1 commented Jan 17, 2025

daosbuild1 commented Jan 17, 2025

daosbuild1 commented Jan 17, 2025

daosbuild1 commented Jan 17, 2025

grom72 commented Jan 21, 2025 • edited Loading

jolivier23 Jan 29, 2025

Choose a reason for hiding this comment

grom72 Jan 30, 2025

Choose a reason for hiding this comment

jolivier23 left a comment

Choose a reason for hiding this comment

jolivier23 commented Jan 29, 2025

jolivier23 Jan 30, 2025

Choose a reason for hiding this comment

grom72 Jan 31, 2025

Choose a reason for hiding this comment

grom72 commented Jan 31, 2025

grom72 commented Aug 13, 2024 •

edited

Loading

github-actions bot commented Aug 13, 2024 •

edited

Loading

grom72 commented Jan 21, 2025 •

edited

Loading